矩阵求导

Matrix Calculus

一、标量方程对向量的导数

分子： $f (\vec{y})$ 为 $1 \times 1$ 的标量
分母： $\vec{y}$ 为 $n \times 1$ 的向量

分母布局

与分母的函数相同

\begin{array}{r} \frac{\partial f (\vec{y})}{\partial \vec{y}} = (\begin{array}{c} \frac{\partial f (\vec{y})}{\partial y_{1}} \\ \frac{\partial f (\vec{y})}{\partial y_{2}} \\ ⋮ \\ \frac{\partial f (\vec{y})}{\partial y_{n}} \end{array}) \end{array}

分子布局

与分子的行数相同

\begin{array}{r} \frac{\partial f (\vec{y})}{\partial \vec{y}} = (\begin{array}{c} \frac{\partial f (\vec{y})}{\partial y_{1}}, \frac{\partial f (\vec{y})}{\partial y_{2}}, \dots, \frac{\partial f (\vec{y})}{\partial y_{n}} \end{array}) \end{array}

二、向量方程对向量的导数

$\vec{f} (\vec{y}) = {(\begin{matrix} f_{1} (\vec{y}) \\ f_{2} (\vec{y}) \\ ⋮ \\ f_{n} (\vec{y}) \end{matrix})}_{(n \times 1)}$ $\vec{y} = {(\begin{matrix} y_{1} \\ y_{2} \\ ⋮ \\ y_{m} \end{matrix})}_{(m \times 1)}$

分母布局

\begin{array}{r} \frac{\partial \vec{f} (\vec{y})}{\partial \vec{y}} = {(\begin{array}{c} \frac{\partial \vec{f} (\vec{y})}{\partial y_{1}} \\ \frac{\partial \vec{f} (\vec{y})}{\partial y_{2}} \\ ⋮ \\ \frac{\partial \vec{f} (\vec{y})}{\partial y_{m}} \end{array})}_{(m \times 1)} = {(\begin{array}{c} \frac{\partial f_{1} (\vec{y})}{\partial y_{1}} & \frac{\partial f_{2} (\vec{y})}{\partial y_{1}} & \dots & \frac{\partial f_{n} (\vec{y})}{\partial y_{1}} \\ \frac{\partial f_{1} (\vec{y})}{\partial y_{2}} & \frac{\partial f_{2} (\vec{y})}{\partial y_{2}} & \dots & \frac{\partial f_{n} (\vec{y})}{\partial y_{2}} \\ \dots & \dots & \dots & \dots \\ \frac{\partial f_{1} (\vec{y})}{\partial y_{m}} & \frac{\partial f_{2} (\vec{y})}{\partial y_{m}} & \dots & \frac{\partial f_{n} (\vec{y})}{\partial y_{m}} \end{array})}_{(m \times n)} \end{array}

结论

在分母布局的条件下：

\begin{aligned} \frac{\partial A \vec{y}}{\partial \vec{y}} & = A^{T} \end{aligned}

二次型的求导

\begin{array}{r} \frac{\partial {\vec{y}}^{T} A \vec{y}}{\partial \vec{y}} = A \vec{y} + A^{T} \vec{y} \end{array}

如果 $A$ 为对称阵，则有 $\frac{\partial {\vec{y}}^{T} A \vec{y}}{\partial \vec{y}} = 2 A \vec{y}$

链式法则

标量的链式求导
注意矩阵不满足乘法交换律
使用分母布局：

\begin{array}{r} \frac{\partial J}{\partial \vec{u}} = \frac{\partial \vec{y}}{\partial \vec{u}} \frac{\partial J}{\partial \vec{y}} \end{array}

离散状态空间方程： ${\vec{x}}_{[k + 1]} = A {\vec{x}}_{[k]} + B {\vec{u}}_{[k]}$
代价函数： $J = {\vec{x}}_{[k + 1]}^{T} {\vec{x}}_{[k + 1]}$
$\frac{\partial J}{\partial \vec{u}} = \frac{\partial {\vec{x}}_{[k + 1]}}{\partial {\vec{u}}_{[k]}} \frac{\partial J}{\partial {\vec{x}}_{[k + 1]}} = B^{T} \cdot 2 {\vec{x}}_{[k + 1]}$

实际应用

线性回归

\begin{aligned} \hat{z} & = y_{1} + y_{2} x \\ J & = \sum_{i = 1}^{n} {[z_{i} - (y_{1} + y_{2} x_{i})]}^{2} \end{aligned}

使用矩阵求解

$\vec{z} = (\begin{matrix} z_{1} \\ z_{2} \\ ⋮ \\ z_{n} \end{matrix}) x = (\begin{matrix} 1 & x_{1} \\ 1 & x_{2} \\ ⋮ & ⋮ \\ 1 & x_{n} \end{matrix}) y = (\begin{matrix} y_{1} \\ y_{2} \end{matrix}) \vec{\hat{z}} = \vec{x} \vec{y} = (\begin{matrix} y_{1} + y_{2} x_{1} \\ y_{1} + y_{2} x_{2} \\ ⋮ \\ y_{1} + y_{2} x_{n} \end{matrix})$

\begin{aligned} \vec{z} - \vec{\hat{z}} & = (\begin{array}{c} z_{1} - (y_{1} + y_{2} x_{1}) \\ z_{2} - (y_{1} + y_{2} x_{2}) \\ ⋮ \\ z_{n} - (y_{1} + y_{2} x_{n}) \end{array}) \\ J & = [\vec{z} - \vec{\hat{z}}]^{T} [\vec{z} - \vec{\hat{z}}] \\ = [{\vec{z}}^{T} - {\vec{y}}^{T} {\vec{x}}^{T}] [\vec{z} - \vec{x} \vec{y}] \\ = {\vec{z}}^{T} \vec{z} - {\vec{y}}^{T} {\vec{x}}^{T} \vec{z} - {\vec{z}}^{T} \vec{x} \vec{y} + {\vec{y}}^{T} {\vec{x}}^{T} \vec{x} \vec{y} \\ = {\vec{z}}^{T} \vec{z} - 2 {\vec{z}}^{T} \vec{x} \vec{y} + {\vec{y}}^{T} {\vec{x}}^{T} \vec{x} \vec{y} \end{aligned}

\begin{aligned} \frac{\partial J}{\partial \vec{y}} = - 2 ({\vec{z}}^{T} \vec{x}) + 2 {\vec{x}}^{T} \vec{x} \vec{y} = 0 \\ \Rightarrow \vec{y} = ({\vec{x}}^{T} \vec{x})^{- 1} {\vec{x}}^{T} \vec{z} \end{aligned}

如果没有解析解，使用机器学习的梯度下降

定义初始值 ${\vec{y}}^{*}$
循环迭代 ${\vec{y}}^{*} = {\vec{y}}^{*} - α \nabla$

梯度： $\nabla = \frac{\partial J}{\partial \vec{y}}$
学习率： $α = (\begin{matrix} α_{1} & 0 \\ 0 & α_{2} \end{matrix})$
注意如果没有归一化处理，学习率的选取要结合未知向量各值的数量级
归一化处理后：各个未知数用同一个学习率，使用标量 $α$ 即可